NARSAD Aim 1

Outline

• Project Aims
• Demographics
• Grid Search
  • Coarse search & hyperparameters
  • Model Selection
  • Feature Selection Parameter
  • Refinement
• ECT models within treatment arm
  • Reflection
  • Brooding
  • TCQR
  • HDRS-6
• Ketamine models within treatment arm
  • Reflection
  • Brooding
  • TCQR
  • HDRS-6
• Sleep deprivation models within treatment arm
  • Reflection
  • Brooding
  • TCQR
  • HDRS-6
• Model Performances Across Treatment Arms

1) Project Aims

Aim 1: Predict degree of improvement in ruminative and depressive symptoms: RRS & HDRS-6.

Aim 2: Determine which treatment-specific models predict across treatment arms.

2) Sample Outline

group	n	mean_age	sd_age	n_males
e	26	40.00000	13.74336	12
k	49	38.85714	10.79159	25
s	52	32.48077	11.22408	23

	Df	Sum Sq	Mean Sq	F value	Pr(>F)
group	2	2.583	1.292	11.13	1.876e-05

**scale**       3    7.736     2.579     22.23    1.694e-13 

group:scale 6 1.757 0.2928 2.524 0.02047

Residuals 483 56.03 0.116 NA NA

Table: 2-way ANOVA Table: Treatment-by-Scale Changes

Quitting from lines 51-85 (NARSAD_Aim_1_Summary.Rmd) Error in pander(ph\(`group:scale`[ph\)group:scale[, “p adj”] < 0.05, ], : object ‘ph’ not found Calls: … withCallingHandlers -> withVisible -> eval -> eval -> pander In addition: Warning messages: 1: Removed 13 rows containing non-finite values (stat_boxplot). 2: Removed 13 rows containing non-finite values (stat_boxplot).

3) Model Selection

Goal:Train and test several types of classifiers (Random Forests, Radial SVM, Gradient Boosted Trees) to narrow down which are best

3a) Coarse search & hyperparameters

Performance by classifier and use of baseline symptoms

Result: Gradient boosted trees consistently outperform both random forests and radial SVMs

Performance by classifier and correlation threshold hyperparameter

Result: A correlation threshold between 0.1 and 0.3 looks best. Now a more refined grid search using boosted trees and a strict cutoff can be done.

3b) Refined grid search

Gradient boosted trees have several important parameters to tune:

Gradient Boosted Tree Outline

• nrounds: Controls the maximum number of iterations. For classification, it is similar to the number of trees to grow.
• eta: Step size shrinkage used in update. After each boosting step, we can directly get the weights of new features, and eta shrinks the feature weights to make the boosting process more conservative. Range 0:1.
• max depth: Maximum depth of a tree. Increasing this value will make the model more complex and more likely to overfit. Range 0:Inf.
• colsample by tree: The subsample ratio of columns when constructing each tree. Subsampling occurs once for every tree constructed.
• gamma: Minimum loss reduction required to make a further partition on a leaf node of the tree. The larger gamma is, the more conservative the algorithm will be. Range 0:Inf
• child weight: Minimum sum of instance weight needed in a child. If the tree partition step results in a leaf node with the sum of instance weight less than min_child_weight, then the building process will give up further partitioning. In linear regression task, this simply corresponds to minimum number of instances needed to be in each node. The larger min_child_weight is, the more conservative the algorithm will be. Range 0:Inf.
• subsample: Subsample ratio of the training instances. Setting it to 0.5 means that XGBoost would randomly sample half of the training data prior to growing trees. This will prevent overfitting. Subsampling will occur once in every boosting iteration. Range 0:1.

Because there are so many model-specific parameters and hyperparameters to tune in a brute force grid search, I’ll tune some of the larger knobs first…

Tuning the nrounds first holding other parameters at constant values

Result: Differences are more pronounced for models excluding baseline severity. I’ll restrict the nrounds range to: {50, 100, 500}

Tuning the max depth parameter holding other parameters constant (nrounds set to 100)

Result: Given the above, I restricted this range to 2:6

Tuning the child weight parameter

Result: No clear winner here; I restricted the range to 1:2.

Tuning eta

Result: Biggest effect is for models including baseline symptoms but general trend is strongly downard for all so I set eta to 0.025.

Tuning gamma

Result: No clear effect. Set to 0.3.

Grid search over restricted range of parameters

Performance for RRS and HDRS6 across restricted range of parameters without baseline symptoms

Results: Performance varies a lot across scales, treatments, and parameterization. This suggests that no one parameterization will be optimal for all outcomes and treatments. I’ll outline the models that perform the best within this restricted grid search.

Highest Performing Models by Arm and Symptom Set

	cutoff	baseline	y	nrounds	eta	maxdepth	gamma	colsamp	childweight	subsamp	variable	PvAc Score
hdrs6.perf_e	0.3	FALSE	hdrs6	500	0.025	6	0.3	0.4	1	1	perf_e	0.2727997
rrsb.perf_e	0.1	FALSE	rrsb	100	0.025	2	0.3	0.4	1	1	perf_e	0.5374308
rrsr.perf_e	0.1	FALSE	rrsr	500	0.025	2	0.3	0.4	2	1	perf_e	0.2085911
tcqr.perf_e	0.1	FALSE	tcqr	500	0.025	6	0.3	0.4	2	0.5	perf_e	0.3676297
hdrs6.perf_k	0.3	FALSE	hdrs6	500	0.025	2	0.3	0.4	2	0.5	perf_k	0.0636506
rrsb.perf_k	0.1	FALSE	rrsb	100	0.025	4	0.3	0.4	2	1	perf_k	0.1596519
rrsr.perf_k	0.3	FALSE	rrsr	100	0.025	2	0.3	0.4	2	1	perf_k	0.3104615
tcqr.perf_k	0.1	FALSE	tcqr	100	0.025	2	0.3	0.4	2	1	perf_k	-0.0265675
hdrs6.perf_s	0.1	FALSE	hdrs6	100	0.025	6	0.3	0.4	2	1	perf_s	-0.0618437
rrsb.perf_s	0.3	FALSE	rrsb	500	0.025	2	0.3	0.4	1	1	perf_s	0.7181260
rrsr.perf_s	0.3	FALSE	rrsr	100	0.025	4	0.3	0.4	2	1	perf_s	0.4281529
tcqr.perf_s	0.1	FALSE	tcqr	500	0.025	2	0.3	0.4	2	0.5	perf_s	0.1539772

4) ECT Models, No Baseline Symptoms

4a) RRS: Reflection

## TableGrob (2 x 3) "arrange": 6 grobs
##   z     cells    name           grob
## 1 1 (1-1,1-1) arrange gtable[layout]
## 2 2 (1-1,2-2) arrange gtable[layout]
## 3 3 (1-1,3-3) arrange gtable[layout]
## 4 4 (2-2,1-1) arrange gtable[layout]
## 5 5 (2-2,2-2) arrange gtable[layout]
## 6 6 (2-2,3-3) arrange gtable[layout]

Frequently selected ICA components

4b) RRS: Brooding

## TableGrob (2 x 3) "arrange": 6 grobs
##   z     cells    name           grob
## 1 1 (1-1,1-1) arrange gtable[layout]
## 2 2 (1-1,2-2) arrange gtable[layout]
## 3 3 (1-1,3-3) arrange gtable[layout]
## 4 4 (2-2,1-1) arrange gtable[layout]
## 5 5 (2-2,2-2) arrange gtable[layout]
## 6 6 (2-2,3-3) arrange gtable[layout]

Frequently selected ICA components

4c) Rumination: TCQR

## TableGrob (2 x 3) "arrange": 6 grobs
##   z     cells    name           grob
## 1 1 (1-1,1-1) arrange gtable[layout]
## 2 2 (1-1,2-2) arrange gtable[layout]
## 3 3 (1-1,3-3) arrange gtable[layout]
## 4 4 (2-2,1-1) arrange gtable[layout]
## 5 5 (2-2,2-2) arrange gtable[layout]
## 6 6 (2-2,3-3) arrange gtable[layout]

Frequently selected ICA components

4d) HDRS 6-item scale

## TableGrob (2 x 3) "arrange": 6 grobs
##   z     cells    name           grob
## 1 1 (1-1,1-1) arrange gtable[layout]
## 2 2 (1-1,2-2) arrange gtable[layout]
## 3 3 (1-1,3-3) arrange gtable[layout]
## 4 4 (2-2,1-1) arrange gtable[layout]
## 5 5 (2-2,2-2) arrange gtable[layout]
## 6 6 (2-2,3-3) arrange gtable[layout]

Frequently selected ICA components

5) Ketamine Models, No Baseline Symptoms

5a) RRS: Reflection

## TableGrob (2 x 3) "arrange": 6 grobs
##   z     cells    name           grob
## 1 1 (1-1,1-1) arrange gtable[layout]
## 2 2 (1-1,2-2) arrange gtable[layout]
## 3 3 (1-1,3-3) arrange gtable[layout]
## 4 4 (2-2,1-1) arrange gtable[layout]
## 5 5 (2-2,2-2) arrange gtable[layout]
## 6 6 (2-2,3-3) arrange gtable[layout]

Frequently selected ICA components

5b) RRS: Brooding

## TableGrob (2 x 3) "arrange": 6 grobs
##   z     cells    name           grob
## 1 1 (1-1,1-1) arrange gtable[layout]
## 2 2 (1-1,2-2) arrange gtable[layout]
## 3 3 (1-1,3-3) arrange gtable[layout]
## 4 4 (2-2,1-1) arrange gtable[layout]
## 5 5 (2-2,2-2) arrange gtable[layout]
## 6 6 (2-2,3-3) arrange gtable[layout]

Frequently selected ICA components

5c) RRS: TCRQ

PvAc for this model was < 0 so not worth outlining results

5d) HDRS 6-item Scale

PvAc for this model very low, < 0.1, so not worth outlining results.

6) Sleep Deprivation Models, No Baseline Symptoms

6a) RRS: Reflection

## TableGrob (2 x 3) "arrange": 6 grobs
##   z     cells    name           grob
## 1 1 (1-1,1-1) arrange gtable[layout]
## 2 2 (1-1,2-2) arrange gtable[layout]
## 3 3 (1-1,3-3) arrange gtable[layout]
## 4 4 (2-2,1-1) arrange gtable[layout]
## 5 5 (2-2,2-2) arrange gtable[layout]
## 6 6 (2-2,3-3) arrange gtable[layout]

Frequently selected ICA components

6b) RRS: Brooding

## TableGrob (2 x 3) "arrange": 6 grobs
##   z     cells    name           grob
## 1 1 (1-1,1-1) arrange gtable[layout]
## 2 2 (1-1,2-2) arrange gtable[layout]
## 3 3 (1-1,3-3) arrange gtable[layout]
## 4 4 (2-2,1-1) arrange gtable[layout]
## 5 5 (2-2,2-2) arrange gtable[layout]
## 6 6 (2-2,3-3) arrange gtable[layout]

Frequently selected ICA components

6c) RRS: TCQR

## TableGrob (2 x 3) "arrange": 6 grobs
##   z     cells    name           grob
## 1 1 (1-1,1-1) arrange gtable[layout]
## 2 2 (1-1,2-2) arrange gtable[layout]
## 3 3 (1-1,3-3) arrange gtable[layout]
## 4 4 (2-2,1-1) arrange gtable[layout]
## 5 5 (2-2,2-2) arrange gtable[layout]
## 6 6 (2-2,3-3) arrange gtable[layout]

Frequently selected ICA components

6d) HDRS 6-item Scale

PvAc < 0; results not worth outlining

7) Performance of Models Across Treatment Arms

Coarse grid-search results by classifier

Result: Gradient boosted trees seem to generalize slightly better, on average.

Cross-treatment performance of all models in refined grid search

Results: Inclusion of baseline symptom severity greatly improves model generalizability, of course. However, the distribution of several model performances across treatment arms is consistently above zero despite excluding baseline severity. Also, the models optimized for performance within treatment arms (green diamonds) are not always the best at generalizing.